Skip to content

docs(examples): example 03 must give confidence per question type, not one formula - #1004

Merged
NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
aashish254:fix-example03-confidence-per-type
Oct 7, 2026
Merged

NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
aashish254:fix-example03-confidence-per-type

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

examples/03_reading_the_result.py is the page the repo sends a reader to for the shape of an
answer — its own docstring says "This is the example to come back to when writing your own glue
code" — and it closed its field tour with:

`confidence` is 1 minus normalised entropy -- a scale that moves with the option
count on the same answer.

Agent._decode_answers does not run one formula. It picks by question type
(laya/agent.py:1366-1402):

  • choice and score → round(confidence_from_probs(p, k), 4), i.e. 1 - H(p) / log(k);
  • noul → round(max(float(p[1]), 1.0 - float(p[1])), 4), a probability of the reported side.

The page calls all three types in one predict(), so that sentence is false for the noul row
of the very table printed six lines above it. The page's own example is the counter-proof: a noul
answer reports the same number in confidence and in answer_confidence, and a normalised
entropy of a two-option distribution is never max(p).

Measured with the repo's own functions (no weights loaded):

distribution confidence_from_probs(p, 2) noul's rule / answer_confidence
[0.9, 0.1] 0.5310 0.9000
[0.75, 0.25] 0.1887 0.7500
[0.6, 0.4] 0.0290 0.6000

The page now states the two formulas with their types, calls the scales incomparable, keeps
answer_confidence as the gate field, and adds the one scope note it was missing: a calibration
payload carrying binning_map remaps answer_confidence and leaves confidence alone
(laya/agent.py:1359-1362, laya/onnx_agent.py:686-691).

Lines are 207 insertions / 4 deletions across examples/03_reading_the_result.py and
tests/test_confidence.py. No behaviour changes: every formula, number and remap the gate asserts
is what the code already produces.

Why

A reader gating on confidence at, say, 0.85 gets two different tests out of one threshold: for a
choice it asks whether the distribution is concentrated, and for a noul it asks whether
p(true) is past 0.85 — while a noul at p(true)=0.6 reads a confidence of 0.600 on the page's
own table, and the same mass as a choice would read 0.029. The sentence taught the reader to treat
those as the same quantity.

This is the same defect class as #972, #973, #974 (examples 40, 18, 30) and as #980
(DecisionResult's docstring), on the page a beginner reads first. #980 holds laya/structured.py,
not this file, so the two PRs share no line.

docs/structured.md and docs/questions-and-answers.md still carry the blanket sentence ("confidence is
normalized entropy" with no type split). They are deliberately left out here so this PR stays one
page and one gate; they are the follow-up.

The gate

Five arms in tests/test_confidence.py, weight-free, prefix page-03/:

  1. Ban + seven positive rules, both witnessed. The single-formula sentence is banned, and the
    ban is asserted to fire on the pre-fix wording as it ships on main — a ban with no witness is a
    guess. The page must then name the entropy formula against choice/score, write the
    max(p[1], 1 - p[1]) form, attribute it to noul, call the scales incomparable, keep
    answer_confidence as max(p), name the gate field with its page, and state the remap scope;
    main's page satisfies none of those, so the rules could not have passed before.
  2. Read off the builder, not off a comment. _confidence_exprs walks _decode_answers by AST
    and returns the source of each answer dict's confidence value — all three types must appear,
    choice/score must go through confidence_from_probs, noul must contain
    1.0 - float(p[1]) and must not have switched to entropy underneath the page.
  3. Earned numbers. The pairs the page cites are rebuilt from
    laya.common.confidence_from_probs and matched back into the page's text, so a cited 0.029 or
    0.531 that drifts by one digit fails there. The arm also asserts that a noul's confidence
    equals answer_confidence — the reason one sentence could never cover both.
  4. Remap scope. test_the_remap_reaches_only_answer_confidence reads the apply_binning_map
    call in both builders and holds its input to ans_raw, the raw answer_confidence. Wrapping
    a confidence expression in the map now breaks a test instead of quietly making the page wrong
    again.
  5. ONNX parity. The page describes both paths, so _decode_answers in laya/onnx_agent.py is
    held to the same two expressions (its pre-computed conf_score is resolved to what it is bound
    to, so the arm compares formulas, not variable names).

Mutation harness laya-bench/mutate_example03_confidence.py: 14 mutants, 14 killed, SURVIVORS
[]
, with the M0 baseline green at 139 passed / 0 failed and git status empty after the run. Each
failure is attributable to a named arm — reverting the page (M1) fails 3 arms, un-attributing the
noul formula (M2), dropping "not comparable" (M3), deleting the remap sentence (M4) and swapping
answer_confidence's definition (M7) each fail exactly 1; moving a cited digit by one (M5 0.029 → 0.028, M6 0.531 → 0.532) fails the earned-numbers arm; changing the agents fails M8 (torch noul
→ entropy, 2 arms), M9 (ONNX noul → raw p(true)), M10 (ONNX choice/score off
confidence_from_probs) and M13/M14 (the map put on confidence as well, one builder at a time).
Three self-checks prove the gate itself is not decorative: M11 removes the whitespace collapsing from
_printed_text and the multi-line sentences stop matching (2 arms), M12 blinds the AST scan to all
but the noul dict (2 arms).

tests/test_confidence.py goes 134 → 139 passed, 0 failed; the count was measured against a
pristine a4a8921 copy of the repo, and the same run on the pristine tree still prints 134 / 0.

Neighbours, at the final commit: tests/test_structured_docs.py 116 checks / 0 failed (it holds the
usage keys on this same page, untouched here), tests/test_evals_api.py 48 / 0,
tests/test_agent_gate.py 11 / 0, tests/test_hooks_api.py 453 / 0.
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 → All checks passed!;
python -m compileall -q laya/ tests/ examples/ → clean.

Deconfliction

Run before editing, hunk by hunk over the 70 open PRs:

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

…not one formula

The page that teaches the shape of an answer said "`confidence` is 1 minus
normalised entropy -- a scale that moves with the option count on the same
answer", but `Agent._decode_answers` picks the formula by type: `confidence_from_probs`
for `choice` and `score`, `max(float(p[1]), 1.0 - float(p[1]))` for `noul`. The
page calls all three types in one `predict()`, so the sentence is false for the
`noul` row of its own table, where `confidence` and `answer_confidence` print the
same number -- which a normalised entropy never does (0.9/0.1 reads 0.531 as
entropy, 0.900 as max(p); 0.6/0.4 reads 0.029 against 0.600).
…as the agents build

Four arms, weight-free, over the page the new reader is told to come back to:

  * the ban on the single-formula sentence, witnessed against the wording as it
    ships on main, plus the seven positive rules that page must now satisfy (and
    main's page satisfies none of them);
  * `_decode_answers`'s per-type `confidence` expressions read out of laya/agent.py
    by AST -- all three answer dicts, entropy for choice/score, max(p[1], 1-p[1])
    for noul, and the noul branch must not have switched to entropy underneath the
    page;
  * the numbers the page cites are recomputed from laya.common.confidence_from_probs,
    so a cited 0.029 or 0.531 that drifts fails there rather than silently;
  * the ONNX builder is held to the same two expressions, because the page describes
    both paths and is not free to be true of only one.

tests/test_confidence.py goes 134 -> 138 passed, 0 failed.
…both builders

The page's last sentence -- a calibration payload remaps `answer_confidence` and leaves
`confidence` alone -- is a claim about `_decode_answers`, not about prose. Read the
`apply_binning_map` call in each builder and hold its input to `ans_raw`, the raw
`answer_confidence`, so wrapping a `confidence` expression in the map fails a test
instead of quietly making the page wrong again.
@NandhaKishorM
NandhaKishorM merged commit f09d9ca into NandhaKishorM:main Oct 7, 2026
30 checks passed
NandhaKishorM pushed a commit that referenced this pull request Oct 7, 2026
… its own types

Seven arms in `tests/test_confidence.py`, inserted above the example-28 section so the
hunk lands at pre-image line 399 -- clear of the nine open PRs appending at 603 and of
#1004's section at 589. No weights, no pydantic, no network.

  * a ban on "`confidence` is normalized entropy" and on the sample note that blamed the
    label count alone, each witnessed against the sentence it replaces;
  * `_attributions()`: a formula has to appear in a sentence naming the question types
    that agent code actually uses it for, with backticked type tokens only, so "the
    choice is explicit" cannot be read as a `choice` question;
  * the same map derived from `Agent._decode_answers` and `OnnxAgent._decode_answers`
    with `ast` (one-hop resolution for the ONNX builder's `conf_score`), so the pages are
    held to the builders and not to a transcription;
  * structured.md's sample recomputed: `answer_confidence` = max(p) and `confidence` =
    1 - H(p)/log(k) over the probabilities printed on the next line, with the named option
    count equal to the label count and the field named as the type the compiler gives it;
  * the Q&A page's three sample answers recomputed per type, its noul row shown to be
    `max(p_true, 1 - p_true)` and not the 0.45 its prose cites as the entropy alternative,
    and its three-row table held to those sample blocks.

Against main's pages 6 of the 7 arms fail; the suite goes 134 -> 141 on this branch.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged in 0.3.29.

Example 03 is "reading the result", so giving one formula for confidence across all three question types is wrong in the worst possible place: it is where a reader learns what the field means.

Confirmed on real weights before merging:

triage  type=choice confidence=0.9286  1-H/log(k)=0.9288
sev     type=score  confidence=0.7651  1-H/log(k)=0.7652
urgent  type=noul   confidence=0.7823  1-H/log(k)=n/a

noul returns confidence equal to answer_confidence, with no option vector to take an entropy over, so the single-formula version was not an approximation but a category error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants